Skip to content

Write initial snapshot memory in place - #1927

Open
ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory
Open

ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory

Conversation

@ludfjig

@ludfjig ludfjig commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

New sandboxes skip copying their initial memory. Snapshot::from_env writes the guest code, PEB, init data and page tables in place, and the host populates the rest, mostly the heap, on first guest touch. Sandbox creation gets 33 to 88% faster for small to large heaps.

Benchmarks

Compared to the PR below.

  • MSHV: Linux 6.6 mshv.
  • KVM: Linux 6.17.
Benchmark MSHV KVM
sandboxes/create_initialized/small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
sandboxes/create_initialized/medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
sandboxes/create_initialized/large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
snapshot_files/cold_start_via_evolve/large 38.8 ms → 10.7 ms (−72%) 29.1 ms → 4.24 ms (−85%)

No benchmark regresses.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.

snapshots

Benchmark Size MSHV KVM
call_and_create default 378 µs → 366 µs (−3%) 360 µs → 379 µs (~)
call_and_create small 425 µs → 411 µs (~) 417 µs → 462 µs (~)
call_and_create medium 789 µs → 777 µs (~) 769 µs → 769 µs (~)
call_and_create large 1.84 ms → 1.82 ms (~) 1.85 ms → 1.87 ms (~)
create default 249 µs → 234 µs (−6%) 262 µs → 254 µs (~)
create small 291 µs → 282 µs (~) 324 µs → 313 µs (~)
create medium 606 µs → 612 µs (~) 606 µs → 629 µs (~)
create large 1.68 ms → 1.68 ms (~) 1.7 ms → 1.68 ms (~)
restore default 69 µs → 71.4 µs (~) 22.2 µs → 24.7 µs (~)
restore small 68.7 µs → 73.9 µs (~) 26.7 µs → 23.5 µs (~)
restore medium 72.3 µs → 77.2 µs (~) 30.8 µs → 28 µs (~)
restore large 113 µs → 110 µs (~) 85.2 µs → 61.7 µs (~)
restore_alternate default 78.6 µs → 82.5 µs (~) 71.8 µs → 92.1 µs (~)
restore_alternate small 82.9 µs → 82.2 µs (~) 93.4 µs → 94.4 µs (~)
restore_alternate medium 86.8 µs → 85.6 µs (~) 97.9 µs → 98.8 µs (~)
restore_alternate large 123 µs → 125 µs (~) 158 µs → 129 µs (~)

guest_calls

Benchmark Size MSHV KVM
call default 45.2 µs → 45.4 µs (~) 17.1 µs → 16.9 µs (~)
call small 45.2 µs → 45.2 µs (~) 17.5 µs → 18.0 µs (~)
call medium 45.4 µs → 45.0 µs (~) 17.5 µs → 17.5 µs (~)
call large 44.7 µs → 45.2 µs (~) 17.6 µs → 17.5 µs (~)
call_with_host_function default 75.2 µs → 73.5 µs (~) 34.5 µs → 31.9 µs (~)
call_with_host_function small 74.1 µs → 74 µs (~) 34.7 µs → 33.4 µs (~)
call_with_host_function medium 75.2 µs → 75.7 µs (~) 32.8 µs → 34.2 µs (~)
call_with_host_function large 75.2 µs → 75.8 µs (~) 32.7 µs → 33.6 µs (~)
call_with_restore default 133 µs → 148 µs (~) 62.9 µs → 62.5 µs (~)
call_with_restore small 140 µs → 139 µs (~) 63.8 µs → 71.9 µs (~)
call_with_restore medium 138 µs → 141 µs (~) 67.9 µs → 67.4 µs (~)
call_with_restore large 166 µs → 166 µs (~) 89.1 µs → 89.8 µs (~)
different_thread - 74 µs → 72.3 µs (~) 42.9 µs → 42.3 µs (~)
interrupt_latency - 48.9 µs → 64.5 µs (~) 35.9 µs → 39.2 µs (~)

guest_functions_with_large_parameters

Benchmark Size MSHV KVM
guest_call_with_large_parameters - 163 ms → 172 ms (~) 916 ms → 890 ms (~)

sample_workloads

Benchmark Size MSHV KVM
24K_in_8K_out_c - 56.5 µs → 57.5 µs (~) 27.6 µs → 27.6 µs (~)
24K_in_8K_out_rust - 56.1 µs → 55.3 µs (~) 27.9 µs → 27.8 µs (~)

sandboxes

Benchmark Size MSHV KVM
create_initialized default 1.72 ms → 1.49 ms (−13%) 2.8 ms → 2.83 ms (~)
create_initialized small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
create_initialized medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
create_initialized large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
create_initialized_and_drop default 21.4 ms → 24.3 ms (~) 14.1 ms → 11 ms (~)
create_initialized_and_drop small 28.9 ms → 25.9 ms (~) 13.3 ms → 15.7 ms (~)
create_initialized_and_drop medium 31.5 ms → 28.3 ms (−10%) 24.3 ms → 14.8 ms (−39%)
create_initialized_and_drop large 57.9 ms → 30.7 ms (−47%) 39.6 ms → 12.4 ms (−69%)
sandbox_from_snapshot default 665 µs → 631 µs (−5%) 971 µs → 947 µs (~)
sandbox_from_snapshot small 812 µs → 805 µs (~) 978 µs → 971 µs (~)
sandbox_from_snapshot medium 2.02 ms → 2.04 ms (~) 1.01 ms → 999 µs (~)
sandbox_from_snapshot large 6.06 ms → 6.16 ms (~) 1.22 ms → 1.21 ms (~)

snapshot_files

Benchmark Size MSHV KVM
cold_start_via_evolve default 1.61 ms → 1.58 ms (~) 2.96 ms → 3.05 ms (~)
cold_start_via_evolve small 4.55 ms → 3.22 ms (~) 5.76 ms → 3.14 ms (~)
cold_start_via_evolve medium 12.3 ms → 5.31 ms (~) 11 ms → 3.52 ms (~)
cold_start_via_evolve large 38.8 ms → 10.7 ms (~) 29.1 ms → 4.24 ms (~)
cold_start_via_snapshot default 1.97 ms → 2.01 ms (~) 2.85 ms → 2.73 ms (~)
cold_start_via_snapshot small 8.93 ms → 9.13 ms (~) 8.95 ms → 8.9 ms (~)
cold_start_via_snapshot medium 53.8 ms → 54 ms (~) 50.9 ms → 50.5 ms (~)
cold_start_via_snapshot large 205 ms → 205 ms (~) 194 ms → 196 ms (~)
cold_start_via_snapshot_unverified default 1.4 ms → 1.43 ms (~) 2.2 ms → 1.95 ms (−11%)
cold_start_via_snapshot_unverified small 2.08 ms → 2.21 ms (~) 2.17 ms → 2.3 ms (~)
cold_start_via_snapshot_unverified medium 3.59 ms → 3.52 ms (~) 2.25 ms → 2.1 ms (~)
cold_start_via_snapshot_unverified large 7.73 ms → 7.67 ms (~) 2.59 ms → 2.34 ms (~)
load_snapshot default 701 µs → 703 µs (~) 701 µs → 692 µs (~)
load_snapshot small 6.71 ms → 6.63 ms (~) 6.68 ms → 6.56 ms (~)
load_snapshot medium 50.4 ms → 50.3 ms (~) 48.8 ms → 48.3 ms (~)
load_snapshot large 198 ms → 197 ms (~) 192 ms → 192 ms (~)
load_snapshot_unverified default 112 µs → 116 µs (~) 123 µs → 130 µs (~)
load_snapshot_unverified small 113 µs → 112 µs (~) 119 µs → 116 µs (~)
load_snapshot_unverified medium 114 µs → 123 µs (~) 120 µs → 123 µs (~)
load_snapshot_unverified large 118 µs → 122 µs (~) 125 µs → 127 µs (~)
save_snapshot default 941 µs → 943 µs (~) 43.4 ms → 39.5 ms (~)
save_snapshot small 9.09 ms → 9.11 ms (~) 61.5 ms → 52.2 ms (~)
save_snapshot medium 69.6 ms → 67.7 ms (~) 402 ms → 396 ms (~)
save_snapshot large 271 ms → 266 ms (~) 1.75 s → 1.74 s (~)

@ludfjig ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026
@ludfjig
ludfjig added this pull request to stack #1929 October 12, 2026 20:01
@hyperlight-gh-bot

This comment has been minimized.

Copilot AI balanced review requested due to automatic review settings October 13, 2026 18:56
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 96fab64 to ab775e0 Compare October 13, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Lazy no-reserve allocations need aggregate admission or commitment accounting to prevent deferred host-wide OOM.

1 open finding
What changed in this PR

Optimizes sandbox creation by constructing initial snapshot memory directly in shared memory and relying on lazy page population.

Changes:

  • Writes guest initialization data in place.
  • Freezes shared memory without copying.
  • Documents the performance improvement.
File Description
src/​hyperlight_host/​src/​sandbox/​snapshot/​mod.rs Builds snapshots directly in shared memory.
src/​hyperlight_host/​src/​mem/​shared_mem.rs Restricts the copying helper to tests and HVF.
CHANGELOG.md Documents lazy memory population.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from ab775e0 to 107a858 Compare October 13, 2026 21:01
@hyperlight-gh-bot

This comment has been minimized.

@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 107a858 to aa55340 Compare October 13, 2026 23:00
@hyperlight-gh-bot

This comment has been minimized.

andreiltd
andreiltd previously approved these changes Oct 14, 2026
@hyperlight-gh-bot

This comment has been minimized.

Snapshot::from_env writes the guest code, PEB and init data straight
into shared memory and freezes it. The host populates the remaining
pages on first guest touch, so sandbox creation cost stays independent
of heap size.

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 421a453 to 5631afe Compare October 21, 2026 19:27
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: 5631afe66fdd
Baseline commit: c7396235f55e

kvm / amd (Linux) (🌟 6.07x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 6.07x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 749.74 ns (➖ 1.01x faster)
vec_bytes 586.79 ns (➖ 1.00x slower)
372.86 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 526.34 ns (➖ 1.00x slower)
65536 142.35 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 13.79 ms (🌟 6.07x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
7.73 ns (➖ 1.00x slower) 7.72 ns (➖ 1.00x faster) 7.72 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 118.13 µs (➖ 1.02x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.27 µs (➖ 1.02x slower) 6.98 µs (➖ 1.04x faster)
65536 2.05 µs (➖ 1.05x slower) 1.93 µs (➖ 1.09x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 6.18 µs (➖ 1.05x faster) 6.22 µs (➖ 1.07x faster)
8192 1.05 µs (➖ 1.05x faster) 1.05 µs (➖ 1.01x slower)
262144 26.83 µs (➖ 1.02x faster)
kvm / intel (Linux) (🌟 5.92x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 5.92x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 810.86 ns (➖ 1.00x slower)
vec_bytes 614.59 ns (➖ 1.00x slower)
749.70 µs (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
262144 624.45 ns (➖ 1.05x faster)
65536 166.87 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 14.70 ms (🌟 5.92x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
8.61 ns (➖ 1.02x slower) 8.57 ns (➖ 1.00x slower) 8.49 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 70.10 µs (➖ 1.09x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.05 µs (➖ 1.00x slower) 9.03 µs (➖ 1.02x faster)
65536 2.58 µs (➖ 1.01x slower) 2.57 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.44 µs (➖ 1.01x faster) 8.44 µs (➖ 1.02x faster)
8192 880.07 ns (➖ 1.03x faster) 866.49 ns (➖ 1.00x faster)
262144 34.88 µs (➖ 1.02x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 908.89 ns (➖ 1.02x slower)
vec_bytes 709.07 ns (➖ 1.00x faster)
316.48 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 671.46 ns (➖ 1.07x faster)
65536 188.29 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 65.96 ms (➖ 1.10x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.93 ns (➖ 1.00x faster) 9.93 ns (➖ 1.01x faster) 9.92 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 107.95 µs (➖ 1.04x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.68 µs (➖ 1.04x slower) 9.12 µs (➖ 1.00x faster)
65536 2.63 µs (➖ 1.02x faster) 2.53 µs (➖ 1.01x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.01 µs (➖ 1.04x faster) 7.97 µs (➖ 1.01x faster)
8192 1.36 µs (➖ 1.03x slower) 1.45 µs (➖ 1.10x slower)
262144 34.87 µs (➖ 1.03x faster)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 968.88 ns (➖ 1.02x slower)
vec_bytes 685.63 ns (➖ 1.04x slower)
924.97 µs (➖ 1.25x slower)

payload_allocation

slot_pool_segmented
262144 646.93 ns (➖ 1.01x slower)
65536 169.86 ns (➖ 1.02x faster)

sandboxes

create_initialized_and_drop
medium 77.56 ms (➖ 1.09x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.66 ns (➖ 1.10x slower) 8.82 ns (➖ 1.00x faster) 8.79 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 62.18 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 8.03 µs (➖ 1.00x slower) 7.99 µs (➖ 1.02x faster)
65536 2.31 µs (➖ 1.01x slower) 2.32 µs (➖ 1.02x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 7.56 µs (➖ 1.00x faster) 7.50 µs (➖ 1.02x faster)
8192 932.01 ns (➖ 1.01x faster) 925.93 ns (➖ 1.02x slower)
262144 40.19 µs (➖ 1.03x faster)
hyperv-ws2025 / amd (Windows) (🌟 6.72x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 6.72x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.19 µs (➖ 1.00x faster)
vec_bytes 800.31 ns (➖ 1.00x faster)
1.84 ms (➖ 1.05x faster)

payload_allocation

slot_pool_segmented
262144 794.77 ns (➖ 1.00x faster)
65536 231.35 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 11.81 ms (🌟 6.72x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.85 ns (➖ 1.00x faster) 11.09 ns (➖ 1.10x slower) 10.07 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 1.70 ms (➖ 2.26x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.13 µs (➖ 1.03x faster) 9.18 µs (➖ 1.01x slower)
65536 2.39 µs (➖ 1.00x slower) 2.38 µs (➖ 1.01x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.93 µs (➖ 1.01x slower) 9.04 µs (➖ 1.02x slower)
8192 1.34 µs (➖ 1.01x faster) 1.34 µs (➖ 1.02x slower)
262144 39.36 µs (➖ 1.18x faster)
hyperv-ws2025 / intel (Windows) (🌟 7.17x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 7.17x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.25 µs (➖ 1.03x faster)
vec_bytes 790.03 ns (➖ 1.01x faster)
4.00 ms (➖ 1.05x slower)

payload_allocation

slot_pool_segmented
262144 751.37 ns (➖ 1.00x slower)
65536 216.42 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 17.28 ms (🌟 7.17x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
10.45 ns (➖ 1.01x faster) 10.10 ns (➖ 1.00x faster) 10.00 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 1.00 ms (➖ 1.27x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 8.09 µs (➖ 1.02x slower) 8.16 µs (➖ 1.03x slower)
65536 2.37 µs (➖ 1.02x faster) 2.38 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.58 µs (➖ 1.02x faster) 8.91 µs (➖ 1.01x faster)
8192 1.16 µs (➖ 1.10x faster) 1.10 µs (➖ 1.01x faster)
262144 42.90 µs (➖ 1.13x faster)

Reported by cargo ci bench-report --candidate run:39661853686 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants